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DETAILED ACTION 

Response to Arguments 

1 . The amendment filed 5/1 9/2008 have been entered and made of record. 

2. The application has pending claim(s) 1-6, 8 and 10-12. 

3. Applicant's arguments filed 5/1 9/2008 have been fully considered but they are 
not persuasive. 

The Applicant alleges, "As previously noted by Applicants ..." in page 4, and 
states respectively that the Examiner has failed to provide a sufficient rationale for 
modifying Simpson using the teachings of Loui because Simpson teaches two separate 
and distinct classification processes [FFNN and RNNCSS processes]. The Examiner 
disagrees because the secondary reference Loui teaches a refinement block (see Loui, 
Figure 1) which takes two processes which run in parallel as inputs (see Loui, Figure 1). 
The two processes that are in parallel and that act as inputs to the refinement block are: 
1 . Loui event classifies images using contextual time data [this is similar to Simpsons 
RNNCCS Recurrent Neural Network classification using the time series image 
sequence] and 2. Loui event classifies images using content image data [this is similar 
to Simpsons initial FFNN single {single frame of a sequence of images} classification 
using spectral and textural image content]. Therefore, the Examiner stated that it would 
be obvious to one of ordinary skill in the art at the time the invention was made to 
modify Simpson's method using Loui's teachings by including the generation of the final 
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/ refined image classification after Simpsons FFNN [initial image classification] and 
RNNCCS [Recurrent Neural Network] classifications [Simpsons FFNN and RNNCCS 
processes would run in parallel in order to act as the two inputs to Loui's refined image 
classification block] in order to combine the content image data and contextual image 
time data into the refined event classification (see Loui, Fig. 1, Section - I. Introduction, 
paragraph 1, lines 20-24) . The Examiner has related Loui's two parallel processes 
which act as inputs to Loui's refinement classification block to Simpson's two processes 
and has exemplified how the introduction of Loui's refinement classification block would 
further improve Simpson's overall classification which provides a completely predictable 
result. The obviousness rationale advanced hereinabove is consistent with the criteria 
articulated in KSR International Co. v. Teleflex Inc., 82 USPQ2d 1385 (U.S. 2007). 
Further discussions are provided in the Art Rejection section below. Therefore claims 
1-6, 8 and 10-12 are still not in condition for allowance and are still not patentably 
distinguishable over the prior art references. 

The Applicant alleges, "Applicants previously discussed Loui ..." in pages 4-5, 
and states respectively that firstly there is nothing in Loui that would suggest to one of 
ordinary skill in the art that the two separate and distinct processes of Simpson should 
be sequentially combined as proposed by the Examiner and states respectively that 
secondly it would seem the Examiner could utilize Loui alone to reject the claims based 
on anticipation based on the Examiner's arguments but instead the Examiner attempts 
to redefine the specific teachings of Simpson based on Loui. Firstly , the Examiner 
disagrees because the Examiner doesn't suggest sequentially combining but suggests 
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combining Simpsons two processes in parallel to act as two inputs to Loui's refinement 
classification block and such a combination is completely predictable as has been 
discussed above. Secondly , the Examiner disagrees because Loui's content based 
event classification may take input data also from other blocks and this was the reason 
why the Examiner didn't use the Loui reference alone to reject the claims based on 
anticipation. However Simpsons teaching of the FFNN single classification only uses 
spectral and textural image content as input data as is claimed for the initial content- 
based classification. Further discussions are addressed above. Therefore claims 1-6, 8 
and 10-12 are still not in condition for allowance and are still not patentably 
distinguishable over the prior art references. 

The Applicant alleges, "Applicants submit that the Examiner is relying ..." in page 
5, and states respectively that the Examiner has failed to establish an appropriate 
rationale as to why one skilled in the art would look to Loui in order to modify Simpson 
and that such a combination is relying on hindsight knowledge of Applicant's own 
disclosure. The Examiner disagrees as has been discussed above [the rationale for 
combining the two references has been discussed above]. Further, in response to 
applicant's argument that the examiner's conclusion of obviousness is based upon 
improper hindsight reasoning, it must be recognized that any judgment on obviousness 
is in a sense necessarily a reconstruction based upon hindsight reasoning. But so long 
as it takes into account only knowledge which was within the level of ordinary skill at the 
time the claimed invention was made, and does not include knowledge gleaned only 
from the applicant's disclosure, such a reconstruction is proper. See In re McLaughlin, 
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443 F.2d 1392, 170 USPQ 209 (CCPA 1971). In this case as has been discussed 
above, the Examiner has used the teachings of the secondary reference Loui to suggest 
the motivation to combine the content image data and contextual image time data into 
the refined event classification (see Loui, Fig. 1, Section - I. Introduction, paragraph 1, 
lines 20-24). Therefore claims 1-6, 8 and 10-12 are still not in condition for allowance 
and are still not patentably distinguishable over the prior art references. 



Claim Rejections - 35 USC §112 

4. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

5. Claim 12 is rejected under 35 U.S.C. 112, second paragraph, as being indefinite 
for failing to particularly point out and distinctly claim the subject matter which applicant 
regards as the invention. 

Re Claim 12 at lines 4-5: The claim limitation "a different revised image classification" 
lacks clear antecedent basis because the Applicant has amended claim 1 to read "final 
image classification" from "revised image classification" [this new 35 U.S.C. 1 12 2— 
paragraph issue has arisen because claim 1 was slightly amended by the Applicant] . 
The claim limitation "a different revised image classification" is suggested to be - a 
different final image classification --. 



Appropriate correction is required. 



Application/Control Number: 1 0/71 2,181 Page 6 

Art Unit: 2624 

Claim Rejections - 35 USC § 103 

6. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

7. Claims 1-2, 4-5 and 12 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Simpson ("A recurrent neural network classifier for improved 
retrievals of areal extent of snow cover" - IEEE - vol 39, Oct 2001 , pages 2135-2147, as 
applied in previous Office Action) in view of Loui ("Automatic image event segmentation 
and quality screening for Albuming Applications" - IEEE - July 2000, cited by the 
Applicant in the Information Disclosure Sheet IDS filed on 1 1/13/2003). 

Re Claim 1 : Simpson discloses a method / single image classification using feed- 
forward neural networks (FFNN) and image sequence classification using recurrent 
neural networks (RNNCCS) (see pages 2138-2139, section B. Present Approaches, 
abstract, lines 8-10) for improving scene classification of a sequence of digital images / 
sequence of twelve daylight scenes per day (see page 2139, section 2, paragraph 1) 
comprising the steps of (a) providing a sequence of images / sequence of twelve 
daylight scenes per day captured in temporal succession / temporal sampling into time 
series (see page 2139, section 2 - Image Sequence Classification Using Recurrent 
Neural Networks, first paragraph); (b) classifying each of the images individually / Single 
Image Classification using Feed-Forward Neural Networks (FFNN) based on 
information / spectral and textural contained in the individual image / single image using 
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no feedback to generate an initial content-based image classification / single image 
classification using spectral and textural information as input for each of the images / 
sequence of twelve daylight scenes per day (Feed Forward Neural Network (FFNN) or 
in other words the single FFNN or in other words the three layer Feed Forward Neural 
Network (NNCCS): each image is individually classified [no feedback] using content 
baed information such as spectral and textural information, see page 2139, Fig. 3a, 
abstract, lines 8-10); and (c) generating the respective initial content-based image 
classification / FFNN and a predetermined temporal context model / Recurrent Neural 
Network (RNNCCS) that considers at least the temporal succession / temporal sampling 
into time series of the sequence of images / sequence of twelve daylight scenes per day 
(Recurrent Neural Network (RNNCCS): time series dependent sequence classification 
[feedback], see page 2139, Fig. 3b, section 2, paragraph 1, "time series" shows that the 
temporal succession of the sequence is considered when inputting data into the 
Recurrent Neural Network, abstract, lines 8-10, page 2139, section 2, paragraph 2 "THE 
RNNCCS network lines 6-8, "more accurate than a single FFNN" shows a more 
accurate classification occurs and also that a comparison of the classification of the 
twelve daylight scene images sequence between the two networks [Single FFNN and 
Recurrent Neural Network] is considered and that these networks indeed do classify 
each image in this twelve scene per day sequence); and (d) storing the image 
classifications / FFNN and RNNCCS in a computer readable storage medium / RAM 
(see Simpson, page 2145, section - D. Performance Issues, the Neural Network 



Application/Control Number: 1 0/71 2,181 Page 8 

Art Unit: 2624 

classifications are processed in a computer environment and therefore are stored in the 
RAM or magnetic disk memory). 

However, Simpson fails to specifically suggest that at least two pairs of 
consecutive images in the sequence of images have different elapsed times between 
their capture, generating a final image classification for each image based at least on 
the respective initial content-based image classification and a predetermined temporal 
context model, and storing the final image classifications. 

Loui discloses at least two pairs of consecutive images / adjacent pictures in the 
sequence of images / sequence of chronologically ordered photos having different 
elapsed times / uneven time interval or time difference between their capture (see Loui, 
Fig. 2, section - II. Image Event Segmentation, paragraph 1, lines 12-16, the time 
interval or the time difference between adjacent pictures are uneven as seen in the 
Figure 2 where some adjacent pictures are very close in relative elapsed time whereas 
some adjacent pictures are far spread in relative elapsed time); generating a final image 
classification / refinement (see Loui, Fig. 1 ) for each image based at least on the 
respective initial content-based image classification / classification using image content 
and a predetermined temporal context model / classification using contextual image 
time data (see Loui, section - I. Introduction, paragraph 1 , lines 1 2-1 7 and 20-24, Fig. 1 , 
Loui event classifies images using content image data [this is similar to Simpsons initial 
FFNN single classification using spectral and textural image content] and event 
classifies images using contextual time data [this is similar to Simpsons RNNCCS 
Recurrent Neural Network classification using the time series image sequence] and 
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creates a final / refinement event classification using both event classifications); and 
storing the final / refinement image classification (see Loui, Figs. 4 and 5, image signal 
processing is accomplished by computer systems and the output of Loui's event 
classification is seen in the computer type output in figures 4 and 5 [ this is similar to 
Simpsons image signal processing classifications being stored in the RAM or magnetic 
disk memory of the computer system]). 

Therefore, it would be obvious to one of ordinary skill in the art at the time the 
invention was made to modify Simpson's method using Loui's teachings by including to 
Simpsons hourly or faster temporal sampling the ability to have consecutive images with 
different elapsed times in order to better relate adjacent pictures with clustering (see 
Loui, Fig. 2, Section - II. Image Event Segmentation, paragraph 1, lines 1-4) and by 
including the generation of the final / refined image classification after Simpsons FFNN 
[initial image classification] and RNNCCS [Recurrent Neural Network] classifications in 
order to combine the content image data and contextual image time data into the 
refined event classification (see Loui, Fig. 1, Section - I. Introduction, paragraph 1, lines 
20-24). 

Re Claim 2: Simpson further discloses the information used in step (b) includes pixel 
information / spectral and textural (see page 2138, section 1 - Single Image 
Classification Using Feed-Forward Neural Networks, first paragraph). 
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Re Claim 4: Loui further discloses the pre-determined temporal context model in step 
(c) is independent of elapsed time between consecutive images (see Loui, Section - II. 
Image Event Segmentation, paragraph 1 , lines 8-10, there could be an instance when 
no time interval data is available). 

Re Claim 5: Simpson further discloses the pre-determined temporal context model in 
step (c) is dependent on elapsed time / temporal sampling into time series between 
consecutive images (see Simpson, page 2139, section 2 - Image Sequence 
Classification Using Recurrent Neural Networks, first paragraph, the Recurrent Neural 
Network uses the twelve daylight scenes per day evenly hourly (hourly or faster) spaced 
sequence of images in a time series manner for further improvement in classification, 
the Neural Network clearly is using that specific twelve scenes which are produced 
hourly per daytime elapse in time, it is not using the last scene from the current daytime 
and then the next first scene in the next daytime). 

Re Claim 12 [as best understood by the Examiner]: Loui further discloses wherein the 
predetermined temporal context model in step (c) is dependent on elapsed time 
between consecutive images in the sequence, such that different elapsed times 
between a particular pair of consecutive images produces a different revised image 
classification for a later-captured image of the particular pair of consecutive images (see 
Loui, Fig. 2, section - II. Image Event Segmentation, paragraph 1, lines 12-16, the time 
interval or the time difference between adjacent pictures are uneven as seen in the 
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Figure 2 where some adjacent pictures are very close in relative elapsed time whereas 
some adjacent pictures are far spread in relative elapsed time, the contextual image 
time data is used along with the content image data for the refined classification, Figs. 4 
and 5 show that different elapsed times between particular pair of consecutive images 
produce different event classification results, if a pair of images are real close in elapsed 
time they will most likely [also dependent upon image content data] be classified in the 
same event, if a pair of images are far in elapsed time they will most likely [also 
dependent upon image content data] be classified in different events). 

8. Claim 3 is rejected under 35 U.S.C. 103(a) as being unpatentable over Simpson 
as modified by Loui, and in further view of Tretter et al (US 6,977,679 B2, as applied in 
previous Office Action). The teachings of Simpson as modified by Loui have been 
discussed above. 

However, Simpson as modified by Loui fails to specifically suggest the 
information used in step (b) includes capture-device generated metadata information. 

Tretter, as recited in claim 3 , discloses the information used in step (b) includes 
capture device generated / digital camera metadata / focusing distance information (see 
abstract, lines 6-13). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to further modify Simpson's method, as modified by Loui, 
using Tretter's teachings by including focusing distance metadata to Simpson's step (b) 
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in order to enhance the classification of snow cover by further distinguishing clouds and 
snow cover by identifying the difference in distance between clouds and the snow cover 
from the satellite which captures images. 



9. Claims 6, 8, and 10-11 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Simpson as modified by Loui, and further in view of Huang 
("Integration of multimodal features for video scene classification based on HMM" - 
IEEE - Sept 1999, pages 53-58, as applied in previous Office Action). The teachings of 
Simpson as modified by Loui have been discussed above. 

However, Simpson as modified by Loui fails to specifically suggest that the 
temporal context model is a non-casual model dependent on both a previous image and 
a subsequent image. 

Huang, as recited in claim 8 , discloses the pre-determined temporal context 
model is a non-casual model / discrete ergodic Hidden Markov Model dependent on 
both the previous image and a subsequent image / visited from any state (see page 55, 
section Product HMM, page 56, section SIMULATION RESULTS, second paragraph, 
the discrete ergodic HMM visits any states or images from any state or image which 
makes the model non-causal). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to further modify Simpson's method, as modified by Loui, 
using Huang's teachings by replacing a non-causal discrete ergodic Hidden Markov 
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Model with Simpson's temporal context model in order to give more correlation data 
between all the images of the sequence instead of just a few images (images before the 
current image) to further improve the accuracy of the classification. 

Although Simpson's temporal context model, as modified by Loui, and in further 
view of Huang's modifications teaches a non-causal discrete ergodic Hidden Markov 
Model, it does not specifically disclose, as recited in claim 6 , that the temporal context 
model is the causal Hidden Markov Model dependent on a previous image, and it does 
not specifically disclose, as recited in claim 10 , that the temporal context model is 
imposed using Viterbi algorithm, and it does not specifically disclose, as recited in claim 
11 , that the temporal context model is imposed using a belief propagation algorithm. It 
would have been obvious to one of ordinary skill in the art at the time the invention was 
made though to have such a feature of causality in a HMM model, a Viterbi algorithm, or 
a belief propagation algorithm for a temporal context model because they are just other 
methods of computing the probability for classification of a particular sequence which 
Huang's non-causal Hidden Markov Model is basically accomplishing. 

Conclusion 

10. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
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TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

1 1 . Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Bernard Krasnic whose telephone number is (571) 270- 
1357. The examiner can normally be reached on Mon-Thur 8:00am-4:00pm and every 
other Friday 8:00am-3:00pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Jingge Wu can be reached on (571) 272-7429. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
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USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Bernard Krasnic 
September 9, 2008 



/Samir A. Ahmed/ 

Supervisory Patent Examiner, Art Unit 2624 



